Pre x Probabilities from Stochastic Tree Adjoining Grammars

Language models for speech recognition typically use a probability model of the form Pr(anja1; a2; : : : ; an 1). Stochastic grammars, on the other hand, are typically used to assign structure to utterances. A language model of the above form is constructed from such grammars by computing the pre x probability P w2 Pr(a1 anw), where w represents all possible terminations of the pre x a1 an. The main result in this paper is an algorithm to compute such pre x probabilities given a stochastic Tree Adjoining Grammar (TAG). The algorithm achieves the required computation in O(n6) time. The probability of subderivations that do not derive any words in the pre x, but contribute structurally to its derivation, are precomputed to achieve termination. This algorithm enables existing corpus-based estimation techniques for stochastic TAGs to be used for language modelling.